1. Field
The present disclosure relates generally to web browsing, and more specifically to an enhanced web-browsing program.
2. Background
The World Wide Web has become the preferred research source for all types of information. Rather than visit a library for information, users turn to their computers for retrieving information. However, with this popularity has come information overload. Untold pages of information are stored in web sites. With the rise in Internet use and information searching, automated browsing has become critical.
This automated browsing is called web crawling or spidering. Many sites, such as search engines, use spidering as a means of providing up to date data. In some cases, web crawling is used to automate maintenance tasks on web sites, such as checking links or validating HTML code. However, certain characteristics of the World Wide Web make crawling very difficult, or adversely affect the performance of a web site. These characteristics include: the huge size of the web, the fast rate of change, and dynamic page generation.
The nature of many web crawlers further contributes to the problems noted above. Many web crawlers perform only one or a few tasks at a time, leading to repeated operations in order to accomplish a list of tasks. Repeated crawler operations may be needed to determine if the pages are cached in a search engine index, and these are just a sample of the problems encountered by typical Web crawlers. These actions may cause server problems for the web sites being crawled and slower operations, such as page access, for users.
There is a need for a web crawler that is capable of enhanced operations, including the ability to perform multiple types of searches in parallel.